Memory Efficient Processing of DNA Sequences in Relational Main-Memory Database Systems

Pipeline breaking operators such as aggregations or joins require database systems to materialize intermediate results. In case that the database system exceeds main memory capacities due to large intermediate results, main-memory database systems experience a massive performance degradation due to paging or even abort queries. In our current research on efficiently analyzing DNA sequencing data using main-memory database systems, we often face the problem that main memory becomes scarce due to large intermediate results during hash join and sort-based aggregation processing. Therefore, in this paper, we discuss alternative join and aggregation techniques suited for our use case and compare their characteristics regarding memory requirements during processing. Moreover, we evaluate different combinations of these techniques with regard to overall execution runtime and scalability to increasing amounts of data to process. We show that a combination of invisible join and array-based aggregation increases memory efficiency enabling to query genome ranges that are one order of magnitude larger than using a hash join counterpart in combination with sort-based aggregation.